London Review of Education 

Volume 14, Number 3, November 2016 


DOI: https://doi.org/10.18546/LRE. 14.3.04 


Theory, evaluation, and practice in widening participation: 
A framework approach to assessing impact 

Annette Hayton' 

University of Bath 

Andrew Bengry-Howell 
Bath Spa University 

The English higher education (HE) system is deeply stratified, with younger students from 
more privileged backgrounds comprising the majority of the student population. Over the 
last 15 years considerable investment has been made to widen participation but attempts to 
evaluate these initiatives and demonstrate impact have presented a major challenge for the HE 
sector. This paper explores the development and application of a framework for evaluating 
and researching university-led interventions. Drawing largely on the theoretical work of 
Bourdieu it provides a basis for designing and evaluating programmes and activities to develop 
student cultural capital and habitus, and foster agency and a sense of belonging in HE settings. 
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Introduction 

The English higher education (HE) system is deeply stratified, with younger students from 
more privileged backgrounds comprising the majority of the student population (HESA, 2014). 
Concerns around upskilling the workforce and social mobility have resulted in a number of 
policy interventions over the last 15 years, designed to widen participation in HE. In England, 
an Office for Fair Access (OFFA) has been established to scrutinize the HE sector’s efforts in 
this area. Early, high-profile widening participation (WP) initiatives such as Aimhigher, which ran 
from 2004 to 201 I, were largely concerned with increasing the proportion of younger students 
from under-represented groups progressing to HE. As a result there has been a strong focus on 
developing pre-entry activities such as summer schools and campus visits to engage school and 
college students. Recent data from the Higher Education Funding Council for England (HEFCE) 
(2010; 2013) have shown a steady increase in the rate of progression to HE of under-represented 
groups; however, it is not possible to make direct links between these macro-level changes and 
specific local activity. 

How to evaluate WP initiatives poses a major challenge for the HE sector, and this paper 
explores the development and application of a framework for evaluating and researching 
university-led interventions. Drawing largely on the theoretical work of Bourdieu (Bourdieu, 
1986; Bourdieu and Wacquant, 1992) it provides a basis for designing and evaluating programmes 
and activities to develop student cultural capital and habitus, and foster agency and a sense of 
belonging in HE settings. It actively rejects a‘deficit’ model to explain the low participation and 
attainment of students from social groups under-represented in HE. Our approach does not 
deny the need for institutional change and increased reflexivity on the part of the academy with 
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regard to its own exclusionary practices. However, attempts to avoid a deficit approach can fall 
into the trap of regarding the mere opportunity to engage with education and university study 
as sufficient (Sen, 2005), underestimating the different capabilities that students possess to access 
HE.We regard successful HE as a transformational experience for all students, but recognize that 
the practices of the academy are more culturally resonant with those of the middle-class student 
majority. From this perspective, WP interventions described here are viewed as developing the 
capitals and capacities students need to effectively negotiate the transition into HE and succeed 
as a university student. 

The framework presented in this paper encapsulates this transformative process, 
operationalizing capitals, capacities, and practices that students develop into a set of learning 
outcomes organized under overarching aims. The framework is designed to be accessible to 
policymakers, academics, practitioners, and non-specialists alike, and this paper focuses on its 
development and application to access and entry to HE. 

The framework, initially developed in one university, has been trialled as part of a network 
comprising several institutions commonly known as NERUPI (Network for Evaluating and 
Researching University Participation Interventions). 1 It has been designed to maximize the 
impact ofWP interventions on three levels, by: (I) providing a robust theoretical and evidence- 
based rationale for the types of intervention that are designed and delivered; (2) providing clear 
aims and learning outcomes for interventions, which enables more strategic and reflexive design 
and delivery; and (3) rationalizing and integrating evaluation processes across programmes of 
intervention to improve data quality and utility, and more effectively demonstrate impact using 
a range of indicators; 

As the demand for more rigorous evaluation of the impact of widening participation gathers 
momentum we need to ensure that it is informed by academic research ... In turn, academic 
research ... should not be confined to describing the problems but should start contributing to 
solutions. 

(Whitty et al., 2015:58) 

The need to develop effective approaches for evaluating WP interventions is widely recognized 
across the HE sector,and in government. In a study commissioned by HEFCE to assess the impact 
of Aimhigher, Gorard and Smith (2006) famously criticized the lack of rigour in evaluation of 
outreach activities. Moreover, they claimed that the approaches used had failed to demonstrate 
whether the activities had any impact at all on increasing university progression rates among 
under-represented groups: 

In summary, there is limited evidence about the effectiveness of different pre-entry interventions 
with young people. Much of the research in this area has focused on students’ perceptions of 
interventions, rather than tracking them into HE. 

(Gorard and Smith, 2006: 35) 

Gorard and Smith’s claims jarred with the perceptions of many, particularly practitioners, who 
robustly defended the transformative power ofWP interventions.Arguably, Gorard and Smith 
used narrow methodological criteria to reach their conclusions, but some of their criticisms can 
still be levelled at interventions being made within this field today. Furthermore, because many 
interventions are not overtly informed by theory or research evidence there can appear to be 
little rationale for their design, content, and style of delivery, or understanding of how they might 
contribute to WP in HE. 

The NERUPI Framework utilizes theoretical understandings about WP and factors 
that might hinder or facilitate progression to university to inform practice, and establish the 
criteria against which the impact of interventions are assessed. It integrates three previously 
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disconnected areas in this field: (I) theoretical perspectives and related academic research; (2) 
external monitoring requirements; and (3) evaluation processes to assess the effectiveness of 
institutional or collaborative interventions. 


Theoretical perspectives and related academic research 

In order to understand the rationale for WP policies in the UK, it is important to situate them 
theoretically and politically. Over the last fifty years a growing body of educational research has 
elucidated some of the reasons for disparities in attainment and progression rates to HE between 
different social groups, pointing to the importance of social and cultural factors in shaping 
educational success and progression to HE (Hayton and Paczuska, 2002; Archer et a/., 2003; 
Reay et a/., 2005; Burke, 2012; Reay et a/., 2013; Whitty et a/., 2015).The importance of ‘cultural’ 
factors in socio-economic disadvantage was recognized by the New Labour Government, 
which in 1997 set up a‘Social Exclusion Unit’ to inform policy, as well as launching a number of 
initiatives including Aimhigher. However, New Labour’s understanding of this issue was strongly 
influenced by very particular notions of social and cultural capital associated with middle-class 
values (Gamarnikow and Green, 1999; Gewirtz, 2001), with policies developed to address the 
perceived ‘deficits’ of socially and economically excluded groups. 

In recent years, academic research into the underlying factors affecting educational 
disadvantage and differential participation has become more nuanced. Although the concept of 
‘capitals’ is still employed to explain disparities, the more illuminating studies are theoretically 
informed by a Bourdieusian approach (Bourdieu and Wacquant, 1992). Bourdieu’s fundamental 
recognition of power differentials between social groups enables us to consider cultural 
differences more dispassionately. The inclusion of power allows us to circumvent the trap of 
understanding educational disparities through a ‘deficit model’, where individuals are deemed 
responsible for their perceived failure and lack of certain capacities. Bourdieu’s concept of habitus 
provides a more complex exposition of how cultural background and setting shapes individual 
experience, capacities, practices, and dispositions, and locates privilege, inequality, and agency 
within specific contexts, thus enabling a more in-depth analysis of contributory factors. 

Important insights into how culture operates in maintaining and reproducing educational 
disadvantage are revealed by researchers (Archer et al., 2003; Crazier et a/., 2008; Reay et a/., 
2013; Bathmaker et al., 2013) who have drawn heavily, although not exclusively, on Bourdieu’s 
theoretical approach. Employing predominantly qualitative methodologies they have investigated 
how under-represented groups are constituted and constitute themselves within an ‘alien’ HE 
cultural setting. While research has highlighted previously hidden sociocultural issues, with the 
potential to inform interventions, there is limited awareness of this research within the HE 
sector and its significance for WP. Some WP practitioners do engage with the findings and take 
them into account when developing interventions. Nevertheless, even in these cases, theoretical 
foundations are rarely made explicit in the aims, design, or evaluation of the activities themselves. 
This is partly because traditional approaches to evaluation do not easily lend themselves to 
capturing and assessing the impact of less tangible factors, which are so important in enabling 
personal, and institutional, transformations (Hayton et al., 2015). 


External monitoring requirements 

Monitoring, evaluating, and assessing impact can seem burdensome to higher education 
institutions (HEIs), but they are important aspects of any policy initiative that seeks to change 
existing practice. Without some level of external accountability, WP can be assigned a lower 
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priority within institutions already fully employed delivering their‘core business’ of teaching and 
research. Engaging more fully with this agenda requires a culture change in staff attitudes and 
understandings, institutional strategies, and administrative processes, in particular those relating 
to recruitment, admissions, student experience, and progression. A transitional ‘space’ has to be 
developed within an HEI to nurture and embed new ways of thinking and working. Furthermore, 
this space, which we refer to as the ‘field’ of widening participation (Naidoo, 2004; Whitty et 
o/., 2015) has to extend beyond institutional boundaries, particularly with regard to access and 
outreach, where collaboration with schools and engagement of parents and communities is so 
crucial. 

Until recently the main focus of OFFA and HEFCE monitoring exercises has been collecting 
evidence of activity, requiring HEIs to report expenditure, set and agree output targets, and 
demonstrate how activity meets those targets. For most HEIs, establishing robust internal 
reporting systems, combined with the development of policies, systems,and activities, represented 
a considerable institutional undertaking. However, the call for greater evidence of impact and 
effectiveness has gathered momentum within government and among HE senior managers, 
particularly following the introduction of higher university fees in 2012, when expenditure 
on WP was linked to fee income. OFFA’s guidance for producing Access Agreements (OFFA, 
2015a; OFFA, 2015b) and Monitoring Returns (OFFA/HEFCE, 2014) represent something of 
a step change in target-setting and requirements for monitoring and evaluation. With greater 
acknowledgement of institutional differences and the issues facing a wider range of social 
groups, a strong emphasis on demonstrating impact has moved evaluation far beyond logging the 
number of participants engaged in ‘outreach’ events. Many HEIs now employ staff to evaluate WP 
activities in order to capture and demonstrate impact to OFFA and HEFCE, but also to generate 
information for their own managerial purposes. 


Effectiveness of WP activities and interventions 

As we have seen,WP research and evaluation have been criticized for their lack of rigour. Gorard 
and Smith summarize the issue as follows; 

Those advocating specific interventions often claim success for them, but most interventions 
have had no rigorous evaluation.We encountered no randomised controlled trials or similar.This 
makes it difficult to judge the success, or otherwise, of any attempts to widen participation in 
the short term. 

(Gorard and Smith, 2006: I 16) 

Demonstrating causal links between WP interventions and increased participation of under¬ 
represented groups in HE is challenging.The issue of attribution bedevils many social justice 
initiatives, and attempting to address these through quantitative methods, control groups, and 
longitudinal studies are fraught with methodological issues and contradictions (Copestake, 2014). 
For example, let us consider two typical university outreach activities; a GCSE mathematics 
‘masterclass’ and a ‘return to study workshop’ for prospective mature students. However 
engaging and successful the‘masterclass’ might be, it could only ever be one element contributing 
to an improved grade at GCSE maths.To claim greater impact would negate the work of schools, 
teachers, and other cultural factors. Similarly, a ‘return to study workshop’ might be deemed 
successful in meeting its learning objectives, but its participants may not progress to HE for 
practical reasons, such as finance or caring responsibilities. 

Larger data sets and sample sizes can address some of these issues and the HE Access 
Tracker (HEAT) database, designed to assess the long-term impact ofWP interventions through 
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tracking participants’ progression patterns, is an example of this in practice. However, practical 
and policy imperatives in the fluid and developing field ofWP demand both a more immediate 
and a more nuanced response. 

While randomized controlled trials (RCTs), longitudinal studies, large data sets, and tracking 
participants can contribute to elucidating the overall picture of student participation in HE, they 
cannot inform day-to-day practice at the micro-level. One of the most commonly used ‘measures’ 
of success by practitioners is the much-criticized participant ‘happy sheet’ or evaluation form. 
Despite its limitations as a tool for demonstrating a causal link between interventions and 
progression to university it does have other uses. It can provide a useful means of collecting 
participant data, such as gender, ethnicity, residence in a Low Participation Neighbourhood (LPN), 
or family background in HE, in order to monitor participation from under-represented groups. 
More importantly, it provides practitioners with immediate feedback about the success of the 
intervention itself, which can inform future delivery.While it is difficult to demonstrate a causal 
link between positive student feedback and subsequent progression to HE, most would accept 
that a negative experience would be less likely to encourage progression. More significantly, a 
systematic exploration of the intervention (or process) can provide opportunities for reflexivity 
among participants and providers, leading to increased understandings of the issues affecting 
progression. 

The challenges surrounding WP evaluation are similar to those associated with assessing 
the effectiveness of financial aid to developing countries. In both cases the value of experimental 
methods is limited:‘Experimental black boxes are poorly suited to the evaluation of complicated 
or complex programmes in unstable environments’ (Picciotto, 2012: 223). Although the 
economic and social environment of the UK is relatively stable compared with many developing 
countries, the education system has been the site of considerable change: the introduction 
of academies and free schools outside local authority control, changes in curriculum and 
examinations, the revolution in further education, and increasing HE participation are only a few 
examples of transformation taking place over the last two decades. From this perspective the 
education system can be regarded as a highly unstable context, with WP, itself a relatively new 
area, comprising largely experimental practice and an underdeveloped theoretical base. 

Picciotto’s (2012: 215-16) account of the decade-long debate within the development 
evaluation community resonates strongly with debates concerning WP. Funders wanted 
categorical proof about which programmes were effective, and considerable philanthropic 
resources were invested in obtaining it. However,a large international team with almost unlimited 
funding was unable to identify one definitive approach for evaluating programmes, and reached 
an uneasy consensus that a mixed-methods approach was required. 

To summarize, while research has elucidated some of the factors behind low participation 
and attainment among groups under-represented in HE, its findings are not informing the 
design, evaluation, and monitoring of interventions, or being comprehensively disseminated to 
practitioners. Practitioner research and evaluation has focused on the successful delivery of 
activities, with little direct engagement with research literature, although day-to-day practices 
reflect a tacit knowledge that is evidenced by its findings. 

Monitoring processes, linked to expenditure, have been useful tools in the generation 
and recording of activity, but have had limited engagement with metadata such as HEFCE’s 
analyses of trends in participation (HEFCE, 2010; HEFCE, 2013), with theoretical research, or 
with practitioner evaluations. The framework presented here seeks to overcome this lack of 
connectivity and provide a more coherent basis for assessing the impact ofWP interventions. 
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The NERUPI Framework 

The NERUPI Framework is theoretically underpinned by the work of the French sociologist 
Pierre Bourdieu, and his notions of capitals, field, and habitus. It is important to recognize that 
Bourdieu’s ideas developed over several decades and that definitions have varied over time. In 
An Invitation to Reflexive Sociology, first published in 1992, Bourdieu describes what he sees as his 
basic proposition regarding capital, which ‘presents itself under three fundamental species (each 
with its own subtypes), namely economic capital, cultural capital and social capital’ (Bourdieu and 
Wacquant, 2007: I 19). These capitals are assigned value by the social groups with pre-existing 
status and power in a particular setting - or ‘field’ - who generally attribute higher value to 
qualities and knowledge that reflect their own capacities. In relation to educational ‘success’, 
Bourdieu (1986) regards both social and cultural capital as highly significant.Again writing in 1992, 
he explains social capital as the benefits an individual and group can acquire from an established 
set of relationships (Bourdieu and Wacquant, 2007: I 19). He further refines this in terms of 
education to identify ‘academic capital’ as the capacity to understand the unspoken rules and 
customs within the academy (Bourdieu, 1986). In The forms of capital’ (1986) he explains how 
differences in educational ‘ability’ are largely the result of an investment in developing certain 
forms of cultural capital. In later work he further refines the concept of cultural capital using the 
term ‘intellectual’ or ‘scientific capital’ to encompass specific subject knowledge and expertise. 
In order to avoid confusion, we have adopted the term ‘intellectual capital’ when referring to 
subject knowledge. 

The NERUPI Framework is predicated on a cultural model of widening participation, which 
locates interventions within a contextual field of engagement where student habitus and the 
institutional habituses of school and university intersect. In this respect, the framework’s emphasis 
on students’ habitus and capital is underpinned by an acknowledgement of HEIs’ responsibilities 
to deliver ‘enabling’ interventions, which facilitate institutional reflexivity as well as personal 
change for participants. 

The development of the framework was informed by action research and its capacity to 
‘make a direct contribution to transformative action’ (Kemmis, 2010: 425). It combined a top- 
down theoretical analysis of the aims ofWP interventions with a bottom-up analysis of how 
the design, content, and delivery of interventions meet specific objectives, and contribute to the 
aims of wider programmes of intervention.This dual process combined research and practice 
through an iterative, reflexive analysis of research literature alongside evaluation feedback from 
participants and practitioners. This revealed a number of important areas of resonance; for 
example, while practitioners did not employ the term ‘habitus’, they implicitly understood the 
importance of overcoming participants’ anxieties about‘fitting in’ within a university setting. 


Social and academic capitals 

Although students from lower socio-economic groups may have aspirations to progress to HE, 
they may ‘have less developed capacities to realise them’ (Reay et al., 2005; Bok, 2010: 176).They 
may be unable to access accurate‘hot’ knowledge about HE within their social networks (Ball and 
Vincent, 1998) and consequently may be unable to develop the ‘navigational capacities’ to make 
informed choices about university (Appadurai, 2004).These two factors are addressed through 
the framework’s Progression Curriculum, (Paczuska, 2002) and the first two of the framework’s 
overarching aims, to: 

(I) develop students’ knowledge and awareness of the benefits of higher education and 
graduate employment 
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(2) develop students’ capacity to navigate higher education and graduate employment 
sectors and make informed choices. 


Habitus and identity 

The next strand is concerned with student identity and preparing students for the experience 
of HE. It is theoretically underpinned by Bourdieu’s concept of habitus, and research on how 
sociocultural factors shape students’ perspectives and experience (for example, Archer et a/., 
2012; Bathmaker et at., 2013; Reay et at., 2009). Developing student ‘resilience’ emerged as a 
key concern in the development of this strand of the framework, in acknowledgement of the 
complex challenges that students from under-represented groups can face when accessing and 
experiencing an unfamiliar HE environment. It informs the design and delivery of interventions 
that enable students to anticipate, experience, and reflect upon the challenges of HE within 
a transitional ‘third space’ (Abrahams and Ingram, 2013), from which they can negotiate the 
cultural landscape of the academy. Foregrounding student identity and resilience also opens 
up opportunities for institutional reflection and challenging normative views and attitudes.The 
student identity strand of the framework is encapsulated in its third aim, to: 

(3) develop students’ confidence and resilience to negotiate the challenge of university life 
and graduate progression. 


Intellectual and skills capital 

Early WP policy initiatives such as Aimhigher were largely focused on raising aspirations and 
awareness.The problematic nature of these two aims has been the subject of much discussion, 
but in actuality raising attainment presents more challenges. It has been argued that it is the 
role of schools and colleges to ensure that students achieve good examination results, not HEIs. 
Certainly without engagement through the school curriculum with what Young (2008) describes 
as ‘powerful knowledge’, young people are placed at a serious disadvantage.Young’s formulation 
correlates closely with the ‘facilitating subjects’, such as maths, the sciences, English, history, 
and languages (Russell Group, 2015), valued by selective universities.The competitive nature of 
admissions to undergraduate degree programmes, particularly at selective universities, means 
that good grades in these subjects are crucial. However, the importance of cultural factors in 
explaining the differential attainment rates of particular social groups is often overlooked: 

Resource differences and collective efforts and investments made or not within families become 
translated into individual ‘ability’ ... identities become tied to routes and programmes inscribing 
social barriers and academic boundaries which are constantly re-privileged within education 
policy and schools ... children and their performances are essentialised rather than seen as 
socially, culturally and economically‘made up’. 

(Ball, 2010:162) 

Reay et at. (2013) demonstrated in detail how middle-class parents mobilize their cultural capital 
to engage positively with the school system and ensure the educational success of their children. 
Their study highlighted the differential challenges working-class parents face in supporting their 
children and how this can impact on examination grades and opportunities for progression to HE. 

Slow progress in widening access to the most selective HEIs, combined with increased 
pressure to meet Access Agreement targets, has led some HEIs to develop interventions with 
the explicit aim of raising attainment. Universities are well placed to contextualize subject 
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knowledge, demonstrate how the school curriculum relates to research and careers, and provide 
access to facilities and opportunities to engage with academic staff. 

When developing the NERUPI Framework, subject ‘knowledge’ was differentiated from 
academic ‘skills’ and incorporated under two distinct aims. This acknowledged that certain 
groups of students may have had limited opportunities to develop skills such as essay writing 
and independent research, which are essential for successful university study. In some cases, 
such as mature students returning to study, the development of academic study skills is a major 
part of pre- and post-entry activity. The fourth aim in the framework is concerned with skills 
development and builds on Bourdieu’s taxonomy through the notion of‘skills capital’, which is 
acquired and demonstrated through academic practice. It incorporates WP interventions that 
set out to: 

(4) develop students’ study skills and capacity for academic attainment and successful 
graduate progression. 

The final strand of the framework relates to the knowledge curriculum and developing students’ 
‘intellectual capital’. This strand is multifaceted and incorporates subject-specific interventions 
that extend and contextualize students’ existing knowledge.This process is encapsulated in the 
fifth aim of the framework, to: 

(5) develop students’ understanding by contextualising subject knowledge. 


Practical learning outcomes 

The theory- and evidence-based top-down process of generating overarching aims for WP 
interventions has coincided with a bottom-up analysis and mapping of the programme of 
interventions delivered by the University of Bath’s Widening Participation Outreach Team, which 
includes campus visits, HE talks, subject taster days, GCSE options days, residential summer 
schools, and the university’s bespoke OnTrack to Bath intensive programme for post-16 students. 
The published aims of the interventions did not fully encapsulate the sophisticated thinking 
behind different types of intervention, or the tacit knowledge and expertise that informed their 
design, content, and delivery. 

While similar interventions shared some common aims, these were not always consistent 
and there was no structure to demonstrate how interventions were interrelated, or to indicate 
differential levels of impact that might be high- and low-intensity activities. Interventions were 
‘floating’ as bounded entities with self-referential aims, but without a framework to locate them 
within the context of an overall programme. 

One of the first tasks involved in generating the framework involved formally distinguishing 
between types of intervention and categorizing them as either low, medium, or high intensity. 
Low-intensity interventions were defined as primarily aspirational, for example campus visits. 
Medium-intensity interventions were also broadly aspirational, but incorporated active learning 
elements, for example GCSE options days or subject taster days, designed to develop students’ 
capacity to navigate future progression to HE. High-intensity interventions were defined as 
sustained or residential activities, for example On Track to Bath, tutoring schemes, and summer 
schools, designed to raise attainment by developing skills and intellectual capital. 

The categorization of interventions by ‘intensity’ ran concomitant with a stratification of 
interventions into four outreach levels, based on the year groups at which they were targeted: 
Level 0 (Year 6 and below); Level I (Years 8-9); Level 2 (Years 10-1 I); and Level 3 (post-16); with 
two additional levels; Level 4 (transition) and Level 5 (in HE), which are not discussed here.The 
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existing aims were rationalized to generate a stratified set of level-specific learning outcomes for 
each of the five overarching theoretical aims.This rationalization process entailed further analysis 
of the design and content of interventions to identify additional learning outcomes, which had 
not been reflected among their original aims. 

This process served the dual purpose of mapping a diverse range ofWP provision and 
locating it within a framework where the learning outcomes for interventions are integrated, 
rationalized, stratified, and specifically aligned with the aims of an inclusive programme ofWP 
activity.The design and evaluation of interventions are tailored to match the expected outcomes 
at each level, enabling more effective assessment of impact. Mapping the programme of activity 
also served as a useful planning tool, by highlighting gaps in current provision and informing the 
design of new interventions. 

The framework’s integration of theoretically informed aims with practical learning 
outcomes enables both a macro-analysis of the impact and effectiveness ofWP interventions, 
and a micro-analysis of its individual components. The stratification of learning outcomes by 
level and categorization of interventions into low, medium, and high intensity has provided a 
methodological rationale for more nuanced and meaningful evaluation processes, which are 
attuned to the intensity of an intervention, level of delivery, and expected impact. 

An illustrative example of how the framework has been utilized to assess and demonstrate 
impact is the two Year-12 residential summer schools, in STEM and social sciences, delivered by 
the University of Bath. Both summer schools are categorized as high-intensity interventions in 
terms of their content, delivery, and residential nature, but more importantly so are their learning 
outcomes and expected impact on supporting student progression to a high-tariff university. 

By taking previous practice into account and clearly defining the aims and learning outcomes 
associated with a Level-3 activity, the framework assisted organizers at the macro-level with 
designing the programmes for the summer schools, and at the micro-level with designing 
individual sessions. It provided a conceptual structure for the event, which assisted organizers 
with communicating its purpose to university academics, support staff, and student ambassadors. 
The learning outcomes provided clear criteria against which the overall event and individual 
sessions could be evaluated, through a corresponding set of evaluation questions.These questions 
featured in a four-stage mixed-method evaluation process, which included: (I) pre-event online 
student questionnaire; (2) in-event observation and semi-structured interviews; (3) end-of-event 
interactive student poll and semi-structured reflective group discussion; and (4) post-event 
online student questionnaire. The data generated from students were supported by additional 
data collected from university academics, support staff, and student ambassadors involved in the 
delivery of the event. 

To demonstrate, some end-of-event student data from the 2015 Year-12 STEM Summer 
School will be presented as evidence in support of each of the key Level-3 learning outcomes for 
each of the five aims in the framework. 

For the first aim, to ‘develop students’ knowledge and awareness of the benefits of higher 
education and graduate employment’, the key Level-3 learning outcome is to ‘enable students to 
investigate course and placement options, and social and leisure opportunities at the University of Bath 
and other universities’. In the end-of-event interactive student poll, 98 per cent agreed (74 per cent 
strongly) that attending the event had increased their knowledge about course and placement 
opportunities in HE and opportunities for university graduates.This finding was supported by 
the qualitative data generated from the end-of-event reflective discussion, which demonstrated 
that students had gained a better understanding of HE and opportunities for graduates in their 
subject area, as one explained: 
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It was good to be learning something new. It showed you where you could go if you continued 
doing chemistry and it was really interesting. 

For the second aim, to ‘develop students’ capacity to navigate higher education and graduate 
employment sectors and make informed choices’ , the key Level-3 learning outcome is to ‘enable 
students to evaluate course, student finance and graduate employment opportunities and make informed 
choices that align with personal interests’. In the end-of-event interactive student poll, 74 per cent 
agreed (34 per cent strongly) that attending the summer school had helped them to make an 
informed decision about which course to study.This finding was again supported by qualitative 
data obtained from the end-of-event reflective discussion, where students emphasized how 
useful it had been to explore subject areas, courses, and graduate opportunities with academics 
and students working in these fields. One commented: 

The summer school really strengthened my knowledge of what course I want to do. I wasn’t 
sure if I wanted to take a science or engineering course, but now I definitely know what course 
I want to do. 

For the third aim, to ‘develop students’ confidence and resilience to negotiate the challenge of university 
life and graduate progression’, the key Level-3 learning outcome is to ‘enable students to anticipate 
challenges they will face in higher education and make a successful transition to university'. In the 
end-of-event student poll, 89 per cent agreed (42 per cent strongly) that attending the summer 
school had made them feel more confident in their ability to succeed at university. This was 
supported by qualitative data from the end-of-event reflective discussion, which one student 
encapsulated by saying: 

I had never done anything like this before, and university just seemed really scary. Doing this has 
made me feel much more confident about going and, like, I really want to go to university. 

For the fourth aim, to ‘develop students’study skills and capacity for academic attainment and successful 
graduate progression’, the key Level-3 learning outcome is to ‘enable students to enhance academic 
skills through collaborative projects that develop capacity for critical thinking, independent research and 
self-directed learning’. In the end-of-event student poll, 81 per cent agreed (21 per cent strongly) 
that attending the summer school had given them skills that would help them to succeed in their 
studies.This resonated with qualitative data obtained from the end-of-event reflective discussion, 
where students highlighted new skills they had gained and how the experience had increased 
their motivation to apply themselves to their studies: 

The experience has certainly fuelled my enthusiasm to continue to study hard and get the grades 
that I need for taking a chemistry degree. 

Finally, for the fifth aim, to ‘develop students’ understanding by contextualising subject knowledge’, the 
key Level-3 learning outcomes are to enable students to: (a) ‘situate existing knowledge within wider 
fields of knowledge and apply to other contexts’ ; and (b) ‘situate existing knowledge and interests within 
the context of university degree programmes and academic disciplines’. In the end-of-event student 
poll, 87 per cent agreed (34 per cent strongly) that attending the summer school had increased 
their understanding of their subject area. In the end-of-event discussion, many students iterated 
how they had gained a better understanding of how knowledge gained at school was connected to 
university subject areas: 

The biology project gave you a real insight into how science really works. With GCSEs you don’t 
really go down to the source of the information.The lecturer gave us a real insight into her work. 

You feel much more exposed to the scientific world than you do in a school classroom. 
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By integrating clear theoretical aims with practical learning outcomes, the NERUPI evaluation 
framework has provided a comprehensive structure for locatingWP interventions, which enables 
a more strategic approach to planning, delivering, and evaluating interventions and programmes 
of activity. The structured approach has enabled rationalization of the design and delivery of 
interventions and also provided a structure in which to methodologically locate evaluation 
processes and more effectively assess and demonstrate impact. Nevertheless, it is sufficiently 
flexible to encompass a range of methodological approaches appropriate to the intervention and 
the age group.Through its application in the field of widening participation to higher education, 
it provides a useful tool for making interventions and evaluation more effective, developing 
institutional reflexivity, improving monitoring, and contributing to theoretical understandings 
within the field. 


Note 

I. The Network for Evaluating and Researching University Participation Interventions (NERUPI) 
Framework was initially developed by Annette Hayton, Head of Widening Participation, University of 
Bath, and Andrew Bengry-Howell, Bath Spa University. 


Notes on the contributors 

Annette Hayton is Head of Widening Participation at the University of Bath and has many years of 
experience in managing widening participation activities. Annette is interested in how educational theory 
can be developed and applied in practice to promote positive change within the system. 

Andrew Bengry-Howell is a Senior Lecturer at Bath Spa University and has worked in higher education for 
over ten years at the universities of Bath, Southampton, and Birmingham as a researcher and lecturer. His 
research interests are in youth, identity, and culture, as well as research methodologies. 
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